Data storage cluster with quorum service protection

ABSTRACT

In a protective quorum service, during an initial period of normal operation in which a clustered pair of data storage nodes provide host I/O access to a data storage object and replicate write-type requests to each other, the nodes are first registered to the quorum service. Subsequently, based on the registration and in response to a first auto promote request from a first-requesting node, a success response is returned and the service enters an auto promoted condition, the success response indicating that the first-requesting node is to continue providing the host I/O access to the data storage object without write replication. In response to receiving a subsequent auto promote request from the other node when in the auto promoted condition, a failure response is returned indicating that the other node is to cease providing host I/O access to the data storage object.

BACKGROUND

The invention is related to the field of data storage, and moreparticularly to operation of clustered data storage systems providinghigh availability, distributed host access to data storage objects suchas volumes, files, etc.

SUMMARY

A method is disclosed of providing a quorum service to a clustered pairof data storage nodes to protect against data corruption by failure ofinter-node replication of write-type host I/O requests directed to adata storage object. During an initial period of normal operation inwhich the data storage nodes each provide host I/O access to the datastorage object and each replicate write-type requests to the other node,each of the data storage nodes is registered to the quorum service inresponse to respective registration requests therefrom. Subsequently andbased on both the data storage nodes being registered, in response toreceiving a first auto promote request from a first-requesting datastorage node, a success response is returned and the quorum serviceenters an auto promoted condition, the success response indicating thatthe first-requesting data storage node is to continue providing the hostI/O access to the data storage object without replicating the write-typehost I/O requests to the other data storage node. In response toreceiving a subsequent auto promote request from the other data storagenode when in the auto promoted condition, a failure response is returnedindicating that the other data storage node is to cease providing hostI/O access to the data storage object. The use of the quorum servicehelps avoid so-called “split brain” operation that could lead to datacorruption.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages will beapparent from the following description of particular embodiments of theinvention, as illustrated in the accompanying drawings in which likereference characters refer to the same parts throughout the differentviews.

FIG. 1 is a block diagram of a data processing system with clustereddata storage systems providing access to a stretched volume;

FIG. 2 is a simplified flow diagram of overall operation of a datastorage system using a quorum service to avoid so-called “split brain”condition and data corruption;

FIG. 3 is a flow diagram of operation of the quorum service;

FIG. 4 is a schematic diagram of operation of a data storage system inconnection with the quorum service;

FIG. 5 is a messaging and flow diagram of basic operation of atime-to-live (TTL) mechanism.

DETAILED DESCRIPTION

Overview

Metro cluster is a storage array deployment in which two storage arraystypically deployed in two different data centers or two server roomswithin the same data center cooperate to expose a single “stretched”volume to application hosts, making those hosts and applications runningon them perceive two volumes hosted by the two arrays as a singlevolume. Primary metro cluster use cases are:

1. Increased availability and disaster avoidance.

2. Resource balancing across datacenters.

3. Storage migration.

Metro cluster configurations are popular with customers looking forhighest levels of resiliency against failures including complete failureof one of participating storage arrays or network connectivity failurebetween arrays. Participating storage arrays detect failures based onability to communicate with the other array. Communication failuresindicate either network link or other array failure. To avoid splitbrain situations (when both arrays keep their sides of the stretchedvolume online, without replicating changes), TTL based Polarizationalgorithm can be used. One storage array, designated as “preferred” forthe volume, regularly grants TTL to its peer, designated as“non-preferred”. In such configuration, non-preferred lives at the mercyof preferred, and once its TTL expires, takes its side of the stretchedvolume offline in a process called Polarization.

The following table outlines the effect of using TTL polarization:

Non- Preferred preferred Array Array Link Resolution Alive Alive AliveOnline on both arrays Alive Alive Failed Online only on Preferred AliveFailed Alive Online only on Preferred Failed Alive Alive Offline on botharrays

As the above table shows, polarization does not handle Preferred arrayfailure, because the Non-Preferred always takes its side of thestretched volume offline if unable to communicate with the preferred.This shortcoming can be improved by introducing a third component—quorumservice, so before taking stretched volume offline, the non-preferredarray can attempt to establish quorum with the quorum service. If thequorum is established the non-preferred array can keep the stretchedvolume online knowing the preferred array can no longer establish quorumand will take the stretched volume offline avoiding split brain.

To introduce a quorum service, behavior for three communicatingcomponents needs to be defined—quorum service, enhanced polarizationalgorithm for the preferred array and enhanced polarization algorithmfor the non-preferred array, where each makes decisions based oncommunications with the other two or inability to communicate. Threecomponents working together shall never result in both arrays servicinghost I/O requests to a stretched volume without being able to replicatedata to each other, as this easily leads to data corruption. Soacceptable outcomes are:

1. Preferred and non-preferred arrays can replicate data to each otherand both continue servicing host I/O to the stretched volume. It doesnot really matter if the arrays can communicate with the quorum service.

2. Only one array continues servicing host I/O. The other one takes itsside of the stretched volume offline.

3. None of arrays service host I/O to the stretched volume.

In some embodiments, the following considerations may also beapplicable:

1. If an array is not certain it is safe to continue servicing I/O itbetter take its side of the stretched volume offline.

2. It may be acceptable to request user interaction to resolve cornercase situations leading to both sides of a stretched volume offline,which are otherwise difficult to resolve through algorithms. Theseshould be infrequent occurrences with well documented procedures andclear notifications via user interface.

The solution is the use of a quorum service as generally outlined abovein the Summary, for which an illustrative embodiment is described below.

Description of Embodiments

FIG. 1 shows a data processing system having a plurality of data storagesystems (DSSs) 10 and host computers (HOSTS) 12 functionally coupled byone or more data communications networks 14. The DSSs 10 constitute aclustered storage system 16, also referred to as a “cluster” herein. Thesystem also includes a computerized subsystem referred to as a “quorumservice” 18. The DSSs 10, which are also referred to as “storage nodes”,“nodes”, and “arrays” herein, include respective storage processors(SPs) 20 and physical storage 22. Each storage processor 20 includesprocessors, memory and interface logic enabling it to execute storagerelated functionality using the associated physical storage 22 forunderlying actual data storage, i.e., processing host I/O requests(reads and writes) from the hosts 12. The physical storage 22 may berealized by known secondary storage technology such as magnetic disk,Flash memory, etc.

As shown, the system further includes one or more replication links 24between the DSSs 10, which may be distinct physical links or logicallinks carried via the network 14. For ease of reference the singular“replication link” is used in this description. Also shown is a logicalvolume 26 that can be accessed by the hosts 10 via both the DSSs 10,under normal operating conditions. In one embodiment the cluster 16 is ahigh-availability cluster whose nodes/DSSs 10 reside in differentphysical locations, such as different buildings of a campus or acrosssome other limited geographic area (e.g., tens of miles). The volume 26may be referred to as a “stretched” volume, referring to its logicalpresence on such physically separated DSSs 10. In normal operation, eachDSS 10 services host I/O requests to the volume 26, which includesreplicating write-type requests to the other DSS 10 via the replicationlink 24.

Although the present description is focused on access to a volume 26specifically, it will be appreciated that the disclosed technique may beused with other types of data storage objects such as volume groups,files and file systems, virtual volumes (vVOLs), containers, etc.

The quorum service 18 is used by the DSSs 10 in the event of a failure(e.g., replication failure) having the potential to create a so-called“split brain” problem in which one DSS 10 is modifying its local copy ofthe volume 26 without such modifications also being made on the otherDSS 10, resulting in data corruption. Essentially, the quorum service 18selects one of the DSSs 10 as a surviving node to continue servicinghost I/Os without replication, and stopping the other DSS 10 fromongoing servicing of host I/Os. Details of the quorum service 18 and itsinteraction with the DSSs 10 are provided below. In the illustratedembodiment the DSSs 10 also utilize a logical session referred to as“time to live” or TTL 30 as part of this protection, but in alternativeembodiments some different inter-DSS protection mechanism may be used orthere may be no such mechanism at all, with reliance on just the quorumservice 18. While the present description assumes use of the TTLmechanism 30, those skilled in the art will understand that there willbe differences in operation of such alternative embodiments.

Briefly regarding the TTL mechanism 30, one DSS 10 (“preferred”)regularly grants permission for continued operation to the other DSS 10(“non-preferred”), so that both DSSs 10 provide host access to thevolume 26 as long as this permission stays in effect. Under variousfailure conditions, the TTL mechanism 30 either fails itself (signifyingthe failure) or it is used in a manner that signals a separatelydetected failure from one DSS 10 to the other (i.e., by stopping therequesting or granting of permission). At such a point, the DSSs 10 thenutilize the quorum service 18 to select a survivor, as described morebelow.

Because of the protection function of the quorum service 18, it ispreferably located in a different fault domain than either of the DSSs10, e.g. in a different building or other physical location to improveits availability when there is a failure affecting one or both DSSs 10or the replication link 24.

FIG. 2 outlines overall operation of the operation and use of the quorumservice 18. At 40, during an initial period of normal operation in whichthe data storage nodes 10 each provide host I/O access to a data storageobject (e.g., volume 26) and each replicates write-type requests to theother node 10, each of the data storage nodes 10 is registered to thequorum service 18 in response to respective registration requeststherefrom. At this point the quorum service 18 enters what is referredto as a “2-registered” condition, meaning that both nodes 10 areproperly registered and thus the remaining quorum functionality isenabled. As explained more below, the registration is important forcoherent operation, and therefore if there are problems withregistration the quorum service 18 refrains from providing the key autopromote functionality until they are resolved.

At 42, based on being in the 2-registered condition, and in response toreceiving a first auto promote request from a first-requesting datastorage node 10, the quorum service 18 returns a success response(acceptance) and enters an auto promoted condition. The success responseindicates that the first-requesting data storage node 10 is to continueproviding the host I/O access to the data storage object (volume 26)without replicating the write-type host I/O requests to the other datastorage node 10. Subsequently, in response to receiving an auto promoterequest from the other data storage node 10 when in the auto promotedcondition, the quorum service 18 returns a failure response (rejection)indicating that the other data storage node 10 is to cease providinghost I/O access to the data storage object. By this operation, thefirst-requesting node 10 is selected as the survivor to provide soleaccess to the storage object without replication, avoiding split-brainoperation and attendant potential data corruption.

Although step 40 contemplates respective registration requests beingsent by nodes 10, in alternative embodiments the registration of bothnodes 10 may be done in some other fashion. For example, there may beonly one registration request sent, by either of the nodes 10 forexample, and the request either explicitly or implicitly serves toregister both nodes 10. In such an embodiment it would be up to thenodes themselves to share quorum service identifier and any otherpertinent information, so that they properly use the same quorum servicefor protection.

Quorum Service API and State Transition Diagram

The quorum service 18 supports application programming interface (API)commands as described below. All API commands are preferably handled inidempotent fashion with transactional persistence of state changes.

As noted above, the present description is focused on volumes inparticular, while being more generally applicable to data storageobjects of various types (examples above). The present focus on volumesin particular is reflected in the quorum API, FIG. 3 and relateddescription below. Those skilled in the art will appreciate that inalternative embodiments the <volume id>, for example, may be generalizedto an <object id> or may be substituted with a different object type,e.g., <file id>. The use of the example <volume id> in this descriptionshould not be taken as limiting the applicability of the disclosedtechnique.

Quorum API commands:

1. “get uuid”—returns quorum service uuid generated by the quorum uponinitialization. The uuid shall never change unless the quorum service iscompletely replaced.

2. “register <volume id><array id>”—registers an array identified by<array id> to communicate through the quorum service for the stretchedvolume with <volume id>. Receipt of “register <volume id><array id>”with the same <array id> as in “<volume id> auto promoted <array id>”means the volume is back in sync across arrays and shall clear “autopromoted” record.

3. “unregister <volume id><array id>”—remote the array from the list ofarrays registered to communicate through the quorum service for thestretched volume <volume id>.

4. “auto promote <volume id><array id>”

-   -   a. Records “<volume id> auto promoted <array id>” if two arrays        are registered for the stretched volume and it is not yet        promoted.    -   b. Fails if just one array is registered or the volume is        already promoted by a different array (but not by the same        array—idempotency).

5. “auto promoted <volume id><array id>”—periodic “auto promoted <volumeid><array id>” invocation by the array that has successfully promotedthe volume is allowed, to restore “<volume id> auto promoted <array id>”if quorum service persistence is lost.

6. “manual promote <volume id><array id>”—allows to record “<volume id>manually promoted <array id>” over “auto promote” or “manual promote”.This API command allows a storage administrator to override automateddecisions.

7. “manual promoted <volume id><array id>”—periodic “manually promoted<volume id><array id>” invocation by the array that has successfullypromoted the volume is allowed, to restore “<volume id> manuallypromoted <array id>” if quorum service persistence is lost.

FIG. 3 is a state transition diagram illustrating operation of thequorum service 18. In normal operation it proceeds from an Empty state50 to a 1-Registered state 52 and then a 2-Registered state 54, at whichpoint it is ready to provide a response to an auto promote request fromone of the arrays 10. Upon receiving such a request, the request isgranted (accepted) and operation proceeds to an Auto Promoted state 56,where the basic function is to then deny (reject) any subsequent autopromote request. Also shown in FIG. 3 is a Manually Promoted state 58which is reached in response to a manual promote operation of anadministrative user. The manual promote functionality enables theadministrative user to exert control in response to certain failureconditions, such as outlined below in the failure analysis section.

A more detailed explanation of FIG. 3 follows after first outlining theuse of the quorum API by the DSSs 10 with reference to FIG. 4 .

FIG. 4 describes key operations of the DSSs 10 in using the quorumservice 18. Operations are shown and described in a sequencecorresponding generally to the timing of their use during a period ofoperations.

1. At 60, each array 10 retrieves quorum service universal identifier(uuid) with “get uuid” as soon as its registered.

2. At 62, each array 10 exchanges quorum service uuid with the peerarray 10 as part of TTL messages, and uses the quorum service forconflict resolution only if both agree on the quorum service to use (asindicated by exchanging the same uuid). This operation enables thearrays 10 to present quorum service state to the user—if volumes areprotected with an agreed quorum service 18 or not. Note that thisprocess also allows for negotiating/selecting one out of multipleavailable quorum services if required.

3. At 64, each array 10 invokes “register <volume id><array id>” to thequorum service 18 periodically once the quorum service 18 is registeredon the array 10.

4. At 66, each array 10 uses “auto promote <volume id><array id>” forfailure resolution during polarization. If the arrays 10 are configuredwith preferred and non-preferred status per the TTL mechanism 30, thenthe preferred array 10 will normally issue an auto promote immediately,whereas the non-preferred array would do so after a delay (e.g., severalseconds), which biases the process toward success for the preferredarray 10. This auto promote requesting may be retried periodically,keeping the volume 26 offline until successful.

5. At 68, the array 10 that has been auto promoted and keeps thestretched volume 26 online can send an “auto promoted <volume id><arrayid>” to repopulate quorum service state if lost.

6. At 70, an array 10 invokes “manual promote <volume id><array id>” ifa volume 26 was manually promoted on a non-preferred array 10 (byoperation of a storage administrator). This manual promote may also beretried periodically until successful.

7. At 72, an array 10 invokes “manual promoted <volume id><array id>”periodically during polarization once manual promote is successful torepopulate quorum service state if lost.

8. At 74, an array 10 invokes “unregister <volume id><array id>” to thequorum service 18 as part of removing quorum service 18 fromconfiguration or replacing the quorum service 18 with a new one.

Now returning to FIG. 3 , operation is summarized using the followingstate transition table:

Current State Event Next State Empty 50 1-register (1^(st) array)1-1-Registered 52 2-auto promoted (restore) 2-Auto Promoted 563-manually promoted 3-Manually (restore) Promoted 58 4-auto promote ormanual 4-Empty 50 promote (erroneous) 1-Registered 52 1-register (2^(nd)array) 1-2-Registered 54 2-manual promote 2-Manually (1^(st) array)Promoted 58 3-unregister (1^(st) array) 3-Empty 50 2-Registered 541-auto promote (array) 1-Auto Promoted 56 2-manual promote (array)2-Manually Promoted 58 3-unregister (array) 3-1-Registered 52 Auto1-auto promote (different 1-Auto Promoted 56 Promoted 56array-error/reject) 2-auto promote 2-Auto Promoted 56 (samearray-accept) 3-manual promote (array) 3-Manually Promoted 58 4-register(same array) 4-2-Registered 54 5-register (different 5-Auto Promoted 56array-error) Manually 1-manual promote (array) 1-Manually Promoted 58Promoted 58 2-auto promote (error/reject) 2-Manually Promoted 583-register (same array) 3-2-Registered 54 4-register (different array)4-Manually Promoted 58

FIG. 5 illustrates the essential operation of the TTL mechanism 30, inthe form of a combination flow diagram and messaging diagram between apreferred DSS (Pref'd) 10-P and a non-preferred DSS (Non-Pref'd) 10-NP.Preferred and Non-Preferred statuses may be assigned by configurationdata or system management activity preceding the illustrated operation.The assignment may be arbitrary or it may be based on some relevantsubstantive difference between the DSSs 10. Overall, operation is asuccession of short loops based on a short timer (e.g., one second) atthe non-preferred DSS 10-NP. A given iteration starts at 80 where thenon-preferred DSS 10-NP records its local time and sends a TTL requestmessage to the preferred DSS 10-P, requesting a new TTL period ofongoing operation. The duration of the TTL period is typically fixed andis some amount greater than the repetition period of the process, suchas 5 seconds for example. At 82, the preferred DSS 10-P receives andprocesses the TTL request. It records its own local time, updates aprojection of the TTL expiration time, and sends a message back to thenon-preferred DSS 10-NP granting the new TTL. At 84 the non-preferredDSS 10-NP receives the grant message and uses it to update/refresh itsown local TTL. This process iterates at regular intervals (e.g., onesecond) as indicated at Next 86.

Each of the DSSs 10 maintains a TTL expiration time, which isperiodically refreshed and used in conjunction with a system time-of-day(TOD) to monitor for expiration of TTL. Whenever the TOD value is higherthan the expiration time, it means TTL has expired and the DSS 10initiates polarization. From time to time the “TTL expiration time” isupdated (via TTL requests). Thus, under normal circumstances the “TTLexpiration time” is always in the future, and as the TOD clocks isticking, TTL updates keep moving the TTL expiration time ahead so thatit stays in the future. If TTL updates stop, due to a failure,eventually the TOD clock progresses past the TTL expiration timer andpolarization will be triggered.

The preferred DSS 10 uses its TTL expiration timer to track itsprojection of the expiration of the TTL timer at the non-preferred DSS10-NP. When a failure occurs (which may be intermittent), TTL grantrequests (or responses) are not being transferred on the link 24, andone or both arrays 10 see their TTL timer expiring. When an array timerexpires, the array 10 stops servicing I/O requests and starts queuingthem instead, while also invoking the quorum service 18 to resolve whichof the two arrays 10 should survive. The TTL mechanism alone woulddictate that only the preferred DSS 10-P should survive, if any.However, additional functionality involving the quorum service 18 canresult in the non-preferred DSS 10-NP surviving in some cases, asexplained more below. If only one of the arrays 10 has connectivity tothe quorum service 18, it's a strong indication that the other DSS 10has been isolated (e.g., in a data center failure), and the DSS 10 withthe intact quorum connectivity should prevail even if it is thenon-preferred DSS 10-NP. If both DSSs 10 have connectivity to the quorumservice 18, the preferred DSS 10-P should be preferred to win.

Appendix—Failure Modes Analysis

The following presents a detailed description of various failurescenarios, desired outcomes, and how they are or may be handled by somecombination of automated action (e.g., quorum service 18) andpotentially manual action of an administrator. This description isgeneralized and may not apply to all embodiments, especially those thatrely solely on the quorum service and do not support an overridingadministrator control mechanism.

1. Assuming the same quorum service is always configured on both arrays,does not go down, does not lose state, and perfect communication pathsbetween arrays and to the quorum service are available. It is sufficientif:

a. quorum service runs in a separate fault domain.

b. quorum service exposes an API so either array can record taking overthe volume transactionally

-   -   i. “auto promote <volume id><first array id>”—OK, which is        transactionally persisted by the quorum service as “<volume id>        auto promoted <array id>”    -   ii. the second “auto promote <volume id><second array id>”—FAIL        as it's already promoted on the other array.    -   iii. quorum service shall not process two calls for the same        volume concurrently (should be easy to use a mutex based on        input parameters).

c. when polarization is invoked, the preferred array invokes “autopromote <volume id><winner id>” first

d. the loser invokes “auto promote <volume id><loser id>” with certaindelay (the same delay used by arrays to exchange TTL messages shallwork)

-   -   i. if the network link between arrays is broken—the preferred        array auto promotes the volume.    -   ii. if the non-preferred array/site goes down—the preferred        array auto promotes the volume.    -   iii. if the preferred array/site goes down—the non-preferred        array auto promotes the volume.

Note: arrays communicate with the quorum service while host I/O is notserviced.

2. If the quorum service does go down for some time:

a. if the link between arrays breaks—neither array can contact thequorum service and both take the volume offline

b. the non-preferred array/site goes down—the preferred cannot contactthe quorum service and takes the volume offline

c. the preferred array/site goes down—the non-preferred cannot contactthe quorum service and takes the volume offline

d. there are two solutions:

-   -   i. both non-preferred and preferred continue trying to “auto        promote <volume id><array id>” while they cannot communicate        with each over during polarization. The first to receive OK from        the quorum service once it's back online takes the volume back        online.    -   Note: it likely makes sense for the preferred to try more        frequently, so if the link between arrays breaks, there is        higher probability the preferred keeps the volume online.    -   ii. admin manually promotes one of the sides if the link breaks,        or the surviving array if one goes down. Before manual promote        it is necessary to make sure the other side is down and will        stay down. Else, the other side may “auto promote <volume        id><array id>” to the quorum service once it's back online and        take the out of sync side of the volume back online as well.    -   iii. as a precaution, the array that was manually promoted shall        periodically attempt to “manual promote <volume id><array id>”.        The quorum service shall just record “<volume id> manually        promoted <array id>” and FAIL subsequent attempts to “auto        promote <volume id><array id>”, but still allow “manual promote        <volume id><array id>” from the same of the other array.

Note: it is assumed that components behave correctly and can trust eachother.

3. If the communication paths to the quorum service experience longdelays, lose packets or fail completely:

a. the link between arrays breaks—if one of arrays has communicationpath to the quorum service it “auto promote <volume id><array id>”—OKand takes the volume online. The other side either can't communicate or“auto promote <volume id><array id>”—FAIL after number of retries, soits side of the volume stays offline.

-   -   i. an interesting corner case is when the preferred array        invokes “auto promote <volume id><array id>” to the quorum        service, which is recorded on the quorum service, but then the        response is not delivered, so FAIL. Unless the preferred retries        “auto promote <volume id><array id>” and receives OK, neither        array will take the volume back online. So “auto promote <volume        id><array id>” shall be idempotent.

b. the non-preferred array/site goes down—if the preferred cancommunicate with the quorum service it auto promotes the volume. If thepreferred cannot communicate with the quorum service the only option ismanual promote of the preferred side. Admin shall make sure thenon-preferred side is down and will stay down.

c. the preferred array/site goes down—if the non-preferred array cancommunicate with the quorum service it auto promotes the volume. If thenon-preferred cannot communicate with the quorum service the only optionis manual promote of the non-preferred side. Admin shall make sure thepreferred side is down and will stay down.

d. network delays or packet losses between non-preferred, preferred andthe quorum service may either delay the quorum service receiving therequest or an array receiving the response:

-   -   i. if “auto promote <volume id><array id>” is delivered to the        quorum service with a delay, it's only handled once delivered.    -   ii. if “auto promote <volume id><array id>” request is lost, the        state of overall system does not change.    -   iii. if response for “auto promote <volume id><array id>” is not        delivered to an array it cannot keep or bring the volume online.    -   iv. arrays shall keep retrying “auto promote <volume id><array        id>” until a response is received.

4. If a quorum service is configured on one array and no quorum serviceor a different quorum service is configured on the other array:

a. if the link between arrays fails, several scenarios are possible:

-   -   i. the preferred invokes “auto promote <volume id><array id>”—OK        to the quorum service it is configured to use, and the        non-preferred may send the same request and succeed to a        different quorum service. As the result both arrays can take its        sides of the volume online resulting in data corruption.    -   ii. the preferred may not be configured to use a quorum service        in which case it will take its side of the volume online and the        non-preferred “auto promote <volume id><array id>” to the quorum        service it is configured to use. As the result both arrays could        potentially take their sides of the volume online resulting in        data corruption.

b. so the quorum service shall successfully complete “auto promote<volume id><array id>” only if there is certainty both arrays intend touse it.

c. this is accomplished with arrays periodically sending “register<volume id><array id>” once the quorum service is configured for use.

-   -   i. As soon as the quorum service receives “register <volume        id><preferred id>” it persists “<volume id> registered        <preferred id>”. In this state attempts to “auto promote” shall        fail.    -   ii. Once the quorum service receives “register <volume        id><non-preferred id>” the record is updated to “<volume id>        registered <preferred id><non-preferred id>” and either array        can “auto promote <volume id><array id>”

d. sending “register <volume id><array id>” just once is sufficient tomake sure both arrays intend to leverage the quorum service, howeversending it periodically would allow the quorum service to reply that theother array has not yet registered to discover potential configurationproblems—“register <volume id><preferred id>”—WARNING “the other arrayhas not yet registered” and present an alert to the admin.

e. periodic communication with the quorum service also allows detectingcommunication problems early and warning the storage admin with analert.

5. If the quorum service is removed or replaced with a different one:

a. if the link between arrays breaks, several scenarios are possible:

-   -   i. the preferred is no longer configured to use a quorum service        so takes its side of the volume online and the non-preferred        “auto promote <volume id><loser id>” to the quorum service it        still uses and the preferred used to use. Since the preferred is        still registered for the <volume id> on the quorum service both        arrays can take its sides of the volume online resulting in data        corruption.    -   ii. the preferred “auto promote <volume id><winner id>”—to the        new quorum service it is configured to use which will FAIL (as        the non-preferred is not registered), and the non-preferred        “auto promote <volume id><loser id>” to the quorum service it        still uses and the preferred used to use. As the result        non-preferred takes its side of the volume online which does not        align with user desired configuration.

b. this is solved with arrays unregistering from the quorum service bysending “unregister <volume id><array id>” to the quorum service as apart of quorum service reconfiguration. In this case:

-   -   i. the preferred is no longer configured to use a quorum service        so takes its side of the volume online and the non-preferred        “auto promote <volume id><non-preferred id>” to the quorum        service FAIL as the other array is no longer registered to        communicate through the quorum service.    -   ii. the preferred “auto promote <volume id><preferred id>”—to        the new quorum service it is configured to use which will FAIL        as the non-preferred is not yet registered for the volume, and        the non-preferred “auto promote <volume id><non-preferred id>”        to the quorum service it still uses which FAIL as well. The        volume if offline on both sides which is likely the right        outcome given the inconsistent configuration.

c. however, what if an array can no longer communicate with the quorumservice that needs to be removed from the configuration, while the otherarray can? The following may be possible:

-   -   i. the preferred is no longer configured to use a quorum        service, but it hasn't succeeded with “unregister <volume        id><winner id>” to the quorum service. So it takes its side of        the volume online and the non-preferred “auto promote <volume        id><loser id>” to the quorum service it still uses and the        preferred used to use. Since the preferred is still registered        for the <volume id> on the quorum service both arrays can take        its sides of the volume online resulting in data corruption.

d. such situation will not happen if arrays are in agreement as to whichquorum service both use for a stretched volume or a volume group:

-   -   i. since the same quorum service can potentially be registered        on individual arrays of the stretched pair using different IP        addresses or FQDNs, quorum service shall generate a UUID and        allow retrieving it “get uuid”    -   ii. arrays shall exchange quorum service uuid as part of TTL        messages for polarization algorithm and only use quorum service        for failure resolution if both intend to use the same quorum        service. As soon as quorum service is removed from configuration        or replaced, this is communicated as part of TTL message and        both arrays shall not use the quorum service until consensus is        achieved again.    -   iii. removing quorum service from preferred configuration while        it cannot communicate to the peer and to the quorum service to        make it bring its side of the volume online is a bad idea, and        should perhaps be prohibited. As in such case the non-preferred        may have brought its side of the volume online already if it can        communicate with the quorum service.

e. Another possibility is that the preferred is no longer configured touse a quorum service, which it communicates to the peer array via TTLmessages. So even though it didn't succeed with “unregister <volumeid><winner id>” to the quorum service it can still take its side of thevolume online, as the non-preferred won't attempt “auto promote <volumeid><loser id>” knowing the quorum service configuration is inconsistentand shall not be used. The same applies for the case when non-preferredor preferred are reconfigured to use different quorum services—the pairsimply falls back to using array preference.

6. If the quorum service loses its database or files:

a. if this happens while both arrays can communicate with each other andwith the quorum service, the state will be soon restored via periodic“register <volume id><array id>” messages.

b. losing state while arrays cannot communicate with each other and thevolume is auto promoted on one side appears to be worse. Couple ideas torecover state:

-   -   i. periodic snapshots/backups of quorum service persistence are        prone to recovering old state within RPO. Seemingly maintaining        ever incrementing transaction # with each “register” and “auto        promote” shall allow the quorum service to catch inconsistency        if an array invokes a quorum service API with transaction #,        which is not increment by one. However this may become very        complex very fast. Consider an example where the loser does        “register <volume id><loser id>” with transaction #N,        subsequently the link between arrays is broken and the winner        does “auto promote <volume id><winner id>” with transaction #M.        If the quorum service state is lost and the recovered state only        has non-preferred transaction #N and not preferred transaction        #M, then the loser can succeed with “auto promote <volume        id><loser id>” transaction #N+1 bringing both sides online and        corrupting the volume state.    -   ii. sending periodic “auto promoted <volume id><array id>”        messages by the array that has the volume promoted, so once the        quorum service state is lost and it starts with blank state,        arrays populate it. This requires the quorum service to trust        arrays maintaining proper state. This looks like a more reliable        approach to restoring quorum service state.

7. If a TTL grant request from the non-preferred to the preferred islost or delayed or TTL grant response from the preferred to thenon-preferred is lost or delayed:

a. if at time t both the non-preferred and the preferred estimateloser's TTL as t+5 and TTL grant requests at t+1, t+2, . . . t+5 arelost, both non-preferred and preferred invoke polarization at t+5. Thenon-preferred is guaranteed to invoke polarization ahead of thepreferred and the preferred is guaranteed to attempt auto promote aheadof the non-preferred. So if the preferred can communicate with thequorum service, it takes its side of the volume online and thenon-preferred keeps its side offline.

b. if at time t both the non-preferred and the preferred estimatenon-preferred TTL as t+5 and TTL grant requests at t+1, t+2, . . . t+5are delivered to the preferred but all responses are lost or deliveredwith extreme delays, the non-preferred invokes polarization at t+5 whilethe preferred estimates loser's TTL as t+10. The preferred invokespolarization after the non-preferred but before the non-preferredattempts auto promote the volume with the quorum service. So if thepreferred can communicate with the quorum service, it takes its side ofthe volume online and the non-preferred keeps its side offline. There isno possibility for split brain as well.

c. if at time t both the non-preferred and the preferred estimatenon-preferred TTL as t+5 and TTL grant requests at t+1, t+2, t+5 aredelivered with extreme delays that are becoming progressively worse att+4, t+8, t+12, t+16, t+20 then the non-preferred invokes polarizationat t+6 the latest (if it receives TTL grant sent by the preferred at t+4for the request initiated at t+1) and the preferred extends TTL untilt+9, t+13, t+17, t+21 and finally t+25. So if the preferred onlyreceives read I/Os from the host, it has no way of knowing thenon-preferred invoked polarization at t+6, communicated with the quorumservice to auto promote the volume at t+16 and subsequently startedservicing host Write I/O without replicating to the preferred resultingin split brain.

-   -   i. the preferred shall check the time at which TTL request is        sent and ignore TTL requests (do not update non-preferred TTL        estimate and do not send a response) that are more than 500 ms.        old. In the example above, all TTL grant requests will be        ignored by the preferred and both the preferred and the        non-preferred will invoke polarization at t+5. To measure TTL        requests age, array clocks need to be synchronized and one        method is to use NTP.

The following scenarios may also be taken into account:

1. The link between arrays breaks, the preferred invokes “auto promote<volume id><preferred id>”—OK. However shortly after it goes down due toa rolling failure. In this case the non-preferred can be promotedmanually to bring the volume back online. As part of promote it sends“manual promote <volume id><array id>”.

2. Once both arrays are back online “<volume id> auto promoted <arrayid>” needs to be cleared, so if a failure happens again, the other arraycan promote the volume if required. This can be done with the array thathas the volume promoted once again starting to send periodic “register<volume id><array id>” once it is in sync with the other array and bothhave the volume online.

3. It may happen that one of arrays hosting a stretched volume shall bereplaced. In this case the volume shall be unstretched back to thesurviving array and then stretched again to the new array. To make surequorum service configuration is correct and does not contain more thantwo registered arrays for the same volume the following needs to bedone:

a. when a volume is unstretched, the array hosting the volume“unregister <volume id><the other array id>” and “unregister <volumeid><this array id>” to delete the volume record from the quorum service.b. it may happen the quorum service is not reachable when the volume isunstretched.

This situation can be solved by generating a new unique “stretchedvolume” id on stretching the volume to the new system.

-   -   i. This will result with a left over record with the previous        stretched volume id (with no longer correct arrays ids) and a        new record with the new volume id (with correct array ids).

4. When an array (or rather stretched volume management logic) comesback online different situations are possible and need to be handledwith care:

a. first the array shall try connecting to the other array. Ifsuccessful, ignore the quorum service state, negotiate which one hasactual volume state, resync and start sending periodic “register <volumeid><array id>”

b. if cannot connect to the other array, attempt to auto promote thevolume by sending “auto promote <volume id><array id>” to the quorumservice, there can be several cases:

-   -   i. cannot communicate with the quorum service due to it being        offline or network issues. Keep the volume offline. Retry        periodically. Periodically try connecting to the other array.    -   ii. quorum service has lost its state, so auto promote attempt        fails. Keep the volume offline and notify the admin of the        failure and that manual promote is still an option.    -   iii. the volume was already promoted by the other array, so auto        promote fails. Keep the volume offline. Periodically try        connecting to the other array.    -   iv. the volume was not yet promoted, so auto promote attempt        succeeds. Bring the volume online and start sending periodic        “auto promoted <volume id><array id>”    -   v. the volume was promoted by this array prior to the failure,        so auto promote attempt succeeds. Bring the volume online and        start sending periodic “auto promoted <volume id><array id”

While various embodiments of the invention have been particularly shownand described, it will be understood by those skilled in the art thatvarious changes in form and details may be made therein withoutdeparting from the scope of the invention as defined by the appendedclaims.

What is claimed is:
 1. A method of providing a quorum service to aclustered pair of data storage nodes to protect against data corruptionby failure of inter-node replication of write-type host I/O requestsdirected to a data storage object, comprising: during an initial periodof normal operation in which the data storage nodes each provide hostI/O access to the data storage object and each replicate write-typerequests to the other data storage node, registering each of the datastorage nodes to the quorum service in response to respectiveregistration requests therefrom; and subsequently and based on both thedata storage nodes being registered, (1) in response to receiving afirst auto promote request from a first-requesting data storage node,returning a success response and entering an auto promoted condition,the success response indicating that the first-requesting data storagenode is to continue providing the host I/O access to the data storageobject without replicating the write-type host I/O requests to the otherdata storage node, and (2) in response to receiving a subsequent autopromote request from the other data storage node when in the autopromoted condition, returning a failure response indicating that theother data storage node is to cease providing host I/O access to thedata storage object, wherein the quorum service operates according to astate transition scheme including an Empty state in which no datastorage nodes are registered, a 2-registered state in which both of thedata storage nodes are registered, and an Auto Promoted statecorresponding to the auto promoted condition, and wherein the AutoPromoted state is reached during normal operation from the 2-Registeredstate and not from the Empty state.
 2. The method according to claim 1,wherein the data storage object is one of a volume, volume group, file,file system, virtual volume, or container.
 3. The method according toclaim 1, wherein the data storage nodes employ a time-to-live mechanismby which a preferred one of the data storage nodes regularly grantspermission for ongoing operation to the other data storage node as anon-preferred node, and wherein the first auto promote request is sentby the first-requesting data storage node upon expiration of a time tolive for the non-preferred node.
 4. The method according to claim 3,wherein the first-requesting data storage node is the preferred node,and the subsequent auto promote request is sent by the non-preferrednode after a predetermined delay after expiration of the time to live.5. The method according to claim 3, wherein the first-requesting datastorage node is the non-preferred node, and the first auto promoterequest is sent by the non-preferred node after a predetermined delayafter expiration of the time to live.
 6. The method according to claim1, wherein registering the data storage nodes includes receivingrespective registration requests from the data storage nodes andregistering each of the data storage nodes in response to its respectiveregistration request.
 7. The method according to claim 1, wherein thequorum service is restored to the Auto Promoted state from the Emptystate in response to receiving an auto promoted request from a datastorage node that was auto promoted during normal operation preceding aloss of state at the quorum service.
 8. The method according to claim 1,wherein the state transition scheme further includes a Manual Promotedstate reached in response to receiving a manual promote request from oneof the data storage nodes based on action of an administrative user. 9.The method according to claim 1, wherein the quorum service generates aquorum service identifier and provides it to the data storage nodes inresponse to identifier requests therefrom, the quorum service identifierbeing used by the data storage nodes to confirm mutual use of the samequorum service for protection of the data storage object.
 10. The methodaccording to claim 1, wherein the state transition scheme furtherincludes a 1-registered state in which only one of the data storagenodes is registered, and wherein the Auto Promoted state is reachedduring normal operation from the 2-Registered state and not from the1-registered state.
 11. A quorum service system comprising one or morecomputerized devices executing computer program instructions to causethe computerized devices to provide a quorum service to a clustered pairof data storage nodes to protect against data corruption by failure ofinter-node replication of write-type host I/O requests directed to adata storage object, operation of the quorum service including: Duringan initial period of normal operation in which the data storage nodeseach provide host I/O access to the data storage object and eachreplicate write-type requests to the other data storage node,registering each of the data storage nodes to the quorum service inresponse to respective registration requests therefrom; and subsequentlyand based on both the data storage nodes being registered, (1) inresponse to receiving a first auto promote request from afirst-requesting data storage node, returning a success response andentering an auto promoted condition, the success response indicatingthat the first-requesting data storage node is to continue providing thehost I/O access to the data storage object without replicating thewrite-type host I/O requests to the other data storage node, and (2) inresponse to receiving a subsequent auto promote request from the otherdata storage node when in the auto promoted condition, returning afailure response indicating that the other data storage node is to ceaseproviding host I/O access to the data storage object, wherein the quorumservice operates according to a state transition scheme including anEmpty state in which no data storage nodes are registered, a2-registered state in which both of the data storage nodes areregistered, and an Auto Promoted state corresponding to the autopromoted condition, and wherein the Auto Promoted state is reachedduring normal operation from the 2-Registered state and not from theEmpty state.
 12. The quorum service system according to claim 11,wherein the data storage object is one of a volume, volume group, file,file system, virtual volume, or container.
 13. The quorum service systemaccording to claim 11, wherein the data storage nodes employ atime-to-live mechanism by which a preferred one of the nodes regularlygrants permission for ongoing operation to the other data storage nodeas a non-preferred node, and wherein the first auto promote request issent by the first-requesting data storage node upon expiration of a timeto live for the non-preferred node.
 14. The quorum service systemaccording to claim 13, wherein the first-requesting data storage node isthe preferred node, and the subsequent auto promote request is sent bythe non-preferred node after a predetermined delay after expiration ofthe time to live.
 15. The quorum service system according to claim 13,wherein the first-requesting data storage node is the non-preferrednode, and the first auto promote request is sent by the non-preferrednode after a predetermined delay after expiration of the time to live.16. The quorum service system according to claim 11, wherein registeringthe data storage nodes includes receiving respective registrationrequests from the data storage nodes and registering each of the datastorage nodes in response to its respective registration request. 17.The quorum service system according to claim 11, wherein the quorumservice is restored to the Auto Promoted state from the Empty state inresponse to receiving an auto promoted request from a data storage nodethat was auto promoted during normal operation preceding a loss of stateat the quorum service.
 18. The quorum service system according to claim11, wherein the state transition scheme further includes a ManualPromoted state reached in response to receiving a manual promote requestfrom one of the data storage nodes based on action of an administrativeuser.
 19. The quorum service system according to claim 11, wherein thequorum service generates a quorum service identifier and provides it tothe data storage nodes in response to identifier requests therefrom, thequorum service identifier being used by the data storage nodes toconfirm mutual use of the same quorum service for protection of the datastorage object.
 20. The quorum service system according to claim 11,wherein the state transition scheme further includes a 1-registeredstate in which only one of the data storage nodes is registered, andwherein the Auto Promoted state is reached during normal operation fromthe 2-Registered state and not from the 1-registered state.